1. Field of the Invention
The present invention relates to an image processing apparatus for employing image data obtained by a plurality of image capturing units, and an image processing method therefor.
2. Description of the Related Art
An image processing apparatus and an image processing method have been proposed whereby a focal length and a diaphragm can be altered after an image has been captured. As an example, disclosed in “Light Field Photography with a Hand-Held Plenoptic Camera”, R. Ng, M. Levoy, et al. , Stanford University Computer Science Tech Report CSTR 2005-02, are the configuration of a plenoptic camera that records the status of light fields inside an optical image capturing system, and a development method for subsequently altering a focal length (hereinafter referred to as “refocusing”) . Further, in “Dynamically Reparameterized Light Fields”, Isaksen et al., ACM SIGGRAPH, pp. 297-306 (2000), a method is disclosed whereby a camera array, which includes a plurality of small image capturing units having large depths of field, is employed to capture images, and based on these images, an image with a small or narrow depth of field is generated (hereinafter, this method is referred to as “controlling of the depth of field”). Generally, for a camera array provided by arranging multiple small cameras, the number of pixels for each camera is small and the resolution is low because of the camera size. Furthermore, compared with an ordinary camera that has the same number of pixels, a plenoptic camera has a low resolution.
A known method for increasing resolution is the super-resolution processing in which a plurality of low-resolution images, where a pixel shift has occurred, are synthesized to generate a high-resolution image (e.g., “Super-Resolution Image Reconstruction: A Technical Overview”, S. C. Park, M. K. Park and M. G. Kang, IEEE Signal Proc. Magazine, Vol. 26, 3, P. 21-36 (2003), and Japanese Patent No. 4654887 and No. 3907729). Furthermore, in a “Very High Definition Image Acquisition Method using Multiple Cameras with Different Pixel Apertures”, Takashi Komatsu, Kiyoharu Aizawa and Takahiro Saito, ITE Technical Report, Vol. 17, No. 29, pp. 13-18 (1993), a method is disclosed for changing pixel pitches to effectively obtain images where pixel shifts have occurred, which are required for super-resolution processing.
When an image for a subject that includes a high-frequency component is formed on an image capturing sensor, and sampling is performed for the image using a frequency lower than that for the high-frequency component, the high-frequency component is mixed with the low-frequency component. As a result, so-called folding noise (or aliasing) occurs. The super-resolution processing, which uses a plurality of images where there is a pixel shift, is a process for separating the mixed high-frequency component and recovering an image signal. It should be noted that, generally, a signal recovered at this time is degraded, compared with one that is for an image formed on the image capturing sensor. This occurs because pixel apertures for the individual pixels of the image capturing sensor have finite sizes in order to collect much light.
A phenomenon, where a signal is degraded because the pixel apertures (hereinafter referred to simply as apertures) have finite sizes, is called the aperture effect. As will be later described in detail, the influence of the aperture effect is very strong, depending on a frequency, and may cause the original signal to be attenuated, and buried in noise. For example, in a typical case wherein the aperture is nearly as large as the pixel, a frequency that is about twice the Nyquist frequency is greatly attenuated by the aperture effect. There are also some other frequencies that are greatly attenuated. When the super-resolution processing is performed to recover the high-frequency component of a signal that is attenuated, the attenuated signal is buried in noise and, in actuality, is lost, and therefore, a high quality, high resolution image can not be obtained.